Method and system for assessment of cognitive function based on mobile device usage

ABSTRACT

A system and method that enables a person to unobtrusively assess their cognitive function from mobile device usage. The method records on the mobile device the occurrence and timing of user events comprising the opening and closing of applications resident on the device, the characters inputted, touch-screen gestures made, and voice inputs used on those applications, performs the step of learning a function mapping from the mobile device recordings to measurements of cognitive function that uses a loss function to determine relevant features in the recording, identifies a set of optimal weights that produce a minimum of the loss function, creates a function mapping using the optimal weights, and performs the step of applying the learned function mapping to a new recording on the mobile device to compute new cognitive function values.

RELATED APPLICATION

The present application is a continuation application of and claimspriority to U.S. patent application Ser. No. 15/214,130, filed Jul. 19,2016, and titled “Method and System for Assessment of Cognitive FunctionBased on Mobile Device Usage,” which is a continuation application ofand claims priority to U.S. patent application Ser. No. 14/059,682,filed Oct. 22, 2013, and titled “Method and System for Assessment ofCognitive Function Based on Mobile Device Usage”. The foregoingapplications are hereby incorporated herein by reference in theirentirety.

BACKGROUND

Field of Art

The invention relates generally to computing a person's cognitivefunction and, more specifically, to unobtrusive assessment of cognitivefunction from mobile device usage.

Background Art

An aging society and increasing lifespan has resulted in an increasedprevalence of cognitive decline in society from mild cognitiveimpairment to Alzheimer's disease. Today, one in eight older Americanshas Alzheimer's disease and Alzheimer's is the sixth leading cause ofdeath in the United States [1]. By 2025, the number of Americans age 65and older with Alzheimer's disease is estimated to increase 30% and by2050 that number is expected to triple, barring any breakthroughs toprevent, slow or arrest the disease [1]. Prior to developing Alzheimer'sdisease, patients go through a six-year prodromal phase of cognitivedecline. The societal burden of mental disease in the elderly isstaggering and poised to worsen.

Repeat studies have demonstrated that a healthy lifestyle of moderatephysical activity, good diet, and social interaction can preservecognitive function and reverse cognitive decline [2,3,4]. Several ofthese studies use group comparisons between intervention and controlgroups and rely on tests of cognitive function pre-intervention andpost-intervention to assess the intervention effects on cognitivefunction [5,6]. But for most individuals, knowing that healthy habitspreserve cognitive function is not sufficiently motivating until it istoo late and the damage is well underway. By the time a family member orphysician first discovers signs of cognitive impairment the subject hassuffered substantial atrophy and loss of key structural brain regionsnecessary for memory, learning, executive function, and attention.Neuropsychological tests administered by a neurologist show highspecificity for cognitive impairment but low sensitivity.

The emergence of online tests available through many application vendorssuch as BrainBaseline [7] may help detect signs of impairment earlierbut only if the subject is highly motivated and persistent in testingthemselves repeatedly and regularly for years. Highly motivatedindividuals are more likely to have healthy habits including physicalactivity, diet, and social interaction and least likely to benefit fromclose surveillance of online behavioral tests, further limiting thevalue of those tests to the broader segment of society that wouldbenefit most. In addition, test practice effects are well documented[8,9] whereby the subject develops test taking skills that increasetheir scores but do not transfer well to real world activities andfurther undermine the test's sensitivity and specificity to cognitivedecline.

The introduction of mobile devices and their broad adoption hasrevolutionized how society interacts both with each other and with theirsurroundings. A smartphone today enables a user to make calls, send andreceive emails and text messages, find their location on a map orretrieve directions to a destination point, browse the internet,download and play game applications, and a host of other activities. Inaddition, these smartphones are equipped with accelerometers andgyroscopes that sense the device's acceleration and orientation in3-dimensions. Processing of the acceleration and orientation signalsreveals the user's activity such as whether the person is walking orjogging.

One company that has leveraged the close interaction of an individualwith their mobile device to make behavioral assessments is Ginger.io[10,11]. Ginger.io provides a smartphone application that tracks thenumber and frequency of calls, text messages, and emails sent, and usesthe device's global positioning system (GPS) and accelerometer to inferactivity level. The target population for Ginger.io's application ispatients with chronic diseases such as diabetes, mental disease, andChron's disease. When a patient deviates from their routine calling andtexting patterns, Ginger.io alerts the individual's caregiver tointervene and assess the situation for noncompliance with medications,inappropriate titration of medications, and other factors that mayprecipitate a flare-up of the patient's disease.

In Ginger.io's application, changes in routine calling, texting, orlocation are interpreted as symptoms of disease flare-up requiringinvestigation and notably have high false positives rates. Cognitivedecline due to aging is an insidious process over several years wheresubtle declines in physical activity, diet changes, and socialengagement are causal, not symptomatic. Diagnosing cognitive declinerequires frequent assessment of the higher order cognitive processes ofexecutive function, working memory, episodic memory, and attention.

What is needed is a method and system to assess cognitive function thatis highly sensitive, specific, and unobtrusive to an individual.

-   1. Alzheimer's Association, 2012 Alzheimer's Disease Facts and    Figures. www.alz.org/downloads/facts_figures_2012.pdf-   2. Christopher Hertzog, et al., Enrichment Effects on Adult    Cognitive Development: Can the Functional Capacity of Older Adults    Be Preserved and Enhanced? Association for Psychological Science,    2009, 9(1):1-65-   3. Interview with Nicholas Spitzer, Crosswords don't make you    clever. The Economist, August, 2013,    http://www.economist.com/blogs/prospero/2013/08/quick-study-neuroscience-   4. Gretchen Reynolds, How exercise can help us learn. New York    Times, August 2013,    http://well.blogs.nytimes.com/2013/08/07/how-exercise-can-help-us-learn/?_r=0-   5. Interview with J. Carson Smith, Exercise may be the best medicine    for Alzheimer's disease. Science Daily, July 2013,    http://www.sciencedaily.com/releases/2013/07/130730123249.htm-   6. J. Carson Smith, et al., Interactive effects of physical activity    and APOE-e4 on BOLD semantic memory activation in healthy elders.    Neuroimage, January 2011; 54(1):635-644-   7. www.brainbaseline.com-   8. Ackerman P L, Individual differences in skill learning: An    integration of psychometric and information processing perspectives.    Psychol Bull, 1987, 102:3-27-   9. Healy A F, Wohldmann E L, Sutton E M, Bourne L E, Jr, Specificity    effects in training and transfer of speeded responses. J Exp Psychol    Learn Mem Cognit, 2006, 32:534-546-   10. www.ginger.io-   11. Owen Covington, ‘Virtual nurse’ helps Forsyth Medical Center    track diabetics. The Business Journal, May 2013,    http://www.bizjournals.com/triad/news/2013/05/20/forsyth-medical-center-using-virtual.html

BRIEF SUMMARY OF INVENTION

The invention enables a person to monitor changes in their cognitivefunction in an unobtrusive manner, to view those changes over time, andto evaluate the impact that changes in their social engagement, physicalactivity, learning activity, and diet have on their cognitive functionevaluation. What is needed is a method and system to assess cognitivefunction that is highly sensitive, specific, and unobtrusive to anindividual. Such a method and system would measure and track a person'scognitive function without explicit input or behavioral changes requiredby the subject, such as repeated neuropsychological evaluations andonline tests. Rather, the method and system would use digitally recordedinteractions of an individual with their mobile devices to computeassessments of cognitive function, detect changes in cognitive function,and infer attribution to changes in behavioral activity withoutdisrupting the user's day-to-day activities or their use of mobiledevices.

One embodiment of the present invention is a method for unobtrusivelyrecording an individual's interaction with their mobile device includingapplications opened, inputs typed, gesture patterns used on atouch-screen, and voice input. The method of the present invention caninclude the step of recording data from the mobile device's globalpositioning system (GPS), accelerometer, and gyroscope to infer dailyactivity including the activity intensity, daily mobility includingmethod of travel, and daily social engagement through latitude andlongitude localization of travel destination to a shopping center, amuseum, or a restaurant. This data will provide insight regarding anindividual lifestyle, including their social skills, level of activityand dietary habits. These items can contribute to good health and/orthey can be indicative of a problem. The method of the present inventioncan further include the step of recording data from the mobile device'sphone, email, and texting applications to capture incoming and outgoingcalls, emails and texts generated, length of conversation, length ofmessages, and discrepancies in voice messages and email messages openedversus received, which are used as additional inputs to infer changes insocial engagement.

The method of the present invention can further include the step ofrecording data from a barcode scanning application used to scanpurchased grocery items, food and beverages consumed, and supplementingthat data with nutritional fact information to track diet attributessuch as caloric input, calories from fat, consumed saturated fats, transfats, cholesterol, sugars, protein, vitamins and minerals. Thisinformation is also indicative of lifestyle that is healthy or not. Thisinformation can also correlate to an increase or decline in anindividual's cognitive function. The method of the present invention canfurther include the step of recording data from wearable devices thatmeasure, by way of example, heart-rate, blood oxymetry, bodytemperature, electroencephalogram, and communicate that information toan application resident on the mobile device to infer the user'sphysical activity, activity intensity, and learning activity. Thisinformation related to an individual's biological vitals can explain whythere is a change in cognitive function and why the change is notproblematic and/or this information can be an indicator of a systemicproblem that will have a long term negative impact on cognitivefunction. The method can further include the step of recording the URLsvisited on an Internet browser application, e-book pages read on ane-book application resident on the mobile device, the contentclassification of the material read and its level of complexity, and thelanguage of the content, to further infer learning activity by the user.

The data captured from the user's mobile device by the method of thepresent invention is persisted in the device's storage and furthertransmitted to a cloud computing system to compute cognitive functionfrom the user's interactions and to infer behavioral activityattribution effects on changes in cognitive function. The cognitivefunction assessment and behavioral activity attributions are presentedto the user in a password protected online portal that reveals positiveand negative contributors to trends in cognitive function assessment andestablishes behavioral targets to improve cognitive function. Thosetargets, and any ensuing improvement or decline in cognitive functionare subsequently measured by the method of the present invention,enabling an unobtrusive, closed-loop method and system for assessing andimproving cognitive function from mobile device usage. The cloudcomputing environment allows a user to change mobile devices and/ormobile device service carriers and still have access to previouslyrecorded data.

The system and method as claimed enables a person to unobtrusivelyassess their cognitive function from mobile device usage. The methodrecords on the mobile device the occurrence and timing of user eventscomprising the opening and closing of applications resident on thedevice, the characters inputted, touch-screen gestures made, and voiceinputs used on those applications, performs the step of learning afunction mapping from the mobile device recordings to measurements ofcognitive function that uses a loss function to determine relevantfeatures in the recording, identifies a set of optimal weights thatproduce a minimum of the loss function, creates a function mapping usingthe optimal weights, and performs the step of applying the learnedfunction mapping to a new recording on the mobile device to compute newcognitive function values.

The system and method as claimed enables a person to unobtrusivelyquantify the effect of mobility, physical activity, learning, socialinteraction and diet on cognitive function. The method records on themobile device one of global positioning system longitude and latitudecoordinates, accelerometer coordinates, and gyroscope coordinates, oneof outgoing and incoming phone calls, outgoing and incoming emails, andoutgoing and incoming text messages, one of URLs visited on an internetbrowser application, books read on an e-reader application, games playedon game applications, and the nutritional content of food consumed,performs the step of learning a function mapping from those recordingsto measurements of cognitive function using a loss function to identifya set of optimal weights that produce a minimum for the loss function,uses those optimal weights to create the function mapping, and performsthe step of computing the variance of the cognitive functionmeasurements that is explained by the function mapping to assign anattribution to the effect of mobility, physical activity, learning,social interaction, and diet on measured changes in cognitive function.

These and other advantageous features of the present invention will bein part apparent and in part pointed out herein below.

BRIEF DESCRIPTION OF THE DRAWING

For a better understanding of the present invention, reference may bemade to the accompanying drawings in which:

FIG. 1 illustrates an embodiment of the unobtrusive cognitive functionassessment system configured in accordance with the present invention,

FIG. 2 is a functional description of the system in accordance with oneembodiment of the present invention,

FIG. 3 illustrates an exemplary computing environment of the unobtrusivecognitive function assessment system configured in accordance with oneembodiment of the present invention,

FIG. 4 is the in vivo monitoring module in accordance with oneembodiment of the present invention,

FIG. 5 is the biometric module in accordance with one embodiment of thepresent invention,

FIG. 6 is the transmission module in accordance with one embodiment ofthe present invention,

FIG. 7(a)-(d) is the cognitive function module in accordance with oneembodiment of the present invention,

FIG. 8(a)-(c) is the plan module in accordance with one embodiment ofthe present invention; and

FIG. 9(a)-(c) is representative of data encoded unobtrusively by the invivo monitoring module in accordance with one embodiment of the presentinvention.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that the drawings and detaileddescription presented herein are not intended to limit the invention tothe particular embodiment disclosed, but on the contrary, the intentionis to cover all modifications, equivalents, and alternatives fallingwithin the spirit and scope of the present invention as defined by theappended claims.

DETAILED DESCRIPTION

According to the embodiment(s) of the present invention, various viewsare illustrated in FIG. 1-8 and like reference numerals are being usedconsistently throughout to refer to like and corresponding parts of theinvention for all of the various views and figures of the drawing.

The following detailed description of the invention contains manyspecifics for the purpose of illustration. Any one of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within scope of the invention. Accordingly, thefollowing implementations of the invention are set forth without anyloss of generality to, and without imposing limitations upon, theclaimed invention.

One implementation of the present invention comprises a system andmethod that enables an unobtrusive assessment of a person's cognitivefunction from mobile device usage, which embodiment teaches a novelsystem and method for recording on the mobile device the occurrence andtiming of user events comprising the opening and closing of applicationsresident on the device, the characters inputted and the touch-screengestures used on those applications, the embodiment further includesperforming the step of learning a function mapping from the mobiledevice recordings to measurements of cognitive function that uses a lossfunction to determine relevant features in the recording, identifies aset of optimal weights that produce a minimum of the loss function,creates a function mapping using the optimal weights, the embodimentfurther includes performing the step of applying the learned functionmapping to a new recording on the mobile device to compute a newcognitive function value. For example, a mixed model function is a typeof function containing both fixed and random effects and is used insettings where repeated measurements are made over time on the samesubjects. The mixed model function can be utilized for the learnedfunction mapping, which develops the appropriate attribution for thecognitive function from the repeated measurements. For the cognitivefunction a deep belief network, a machine learning function composed oflayers of neural networks capable of learning complex high-levelfeatures, can be utilized. A software application can reside on a mobilecomputing device, such as a smart phone, personal data assistant (PDA),or tablet computer such that when it is executed by the processor of thecomputing device, the steps of the method are performed.

Another implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect ofphysical activity on cognitive function, which embodiment teaches anovel system and method for repeatedly recording on the mobile deviceone of global positioning system longitude and latitude coordinates,accelerometer coordinates, and gyroscope coordinates, the implementationfurther includes performing the step of learning a function mapping fromthose recordings to measurements of cognitive function using a lossfunction to identify a set of optimal weights that produce a minimum forthe loss function, uses those optimal weights to create the functionmapping, the embodiment further includes performing the step ofcomputing the variance of the cognitive function measurements that isexplained by the function mapping to assign an attribution to the effectof physical activity on measured changes in cognitive function.

A further implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect ofsocial activity on cognitive function, which embodiment teaches a novelsystem and method for repeatedly recording on the mobile device one ofoutgoing and incoming phone calls, outgoing and incoming emails, andoutgoing and incoming text messages, the implementation further includesperforming the step of learning a function mapping from those recordingsto measurements of cognitive function using a loss function to identifya set of optimal weights that produce a minimum for the loss function,uses those optimal weights to create the function mapping, theembodiment further includes performing the step of computing thevariance of the cognitive function measurements that is explained by thefunction mapping to assign an attribution to the effect of socialactivity on measured changes in cognitive function.

A further implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect oflearning activity on cognitive function, which embodiment teaches anovel system and method for repeatedly recording on the mobile deviceone of URLs visited on an Internet browser application, books read on ane-reader application, games played on game applications, theimplementation further includes performing the step of learning afunction mapping from those recordings to measurements of cognitivefunction using a loss function to identify a set of optimal weights thatproduce a minimum for the loss function, uses those optimal weights tocreate the function mapping, the embodiment further includes performingthe step of computing the variance of the cognitive functionmeasurements that is explained by the function mapping to assign anattribution to the effect of learning activity on measured changes incognitive function.

A further implementation of the present invention comprises a system andmethod that enables a person to unobtrusively quantify the effect ofdiet on cognitive function, which embodiment teaches a novel system andmethod for repeatedly recording on the mobile device the nutritionalcontent of food consumed, the implementation further includes performingthe step of learning a function mapping from those recordings tomeasurements of cognitive function using a loss function to identify aset of optimal weights that produce a minimum for the loss function,uses those optimal weights to create the function mapping, theembodiment further includes performing the step of computing thevariance of the cognitive function measurements that is explained by thefunction mapping to assign an attribution to the effect of diet onmeasured changes in cognitive function.

The loss function for the cognitive function can evaluate baselinerecordings and changes and trends in the recordings both from aquantitative and qualitative perspective. For example the frequency ofevents can be monitored for cognitive measurement as well as the qualityof each event, such as latencies between gestures or keystrokes,erroneous keystrokes or gestures, duration of key presses, misspellingsand other activity.

Another implementation of the invention can be a computer systemcomprising a mobile computer (for example a smart-phone; tablet or PDAcomputing device) including a wireless network interface whichcommunicates over a wireless network with a second computer including anetwork interface, each computer further including a processor, a memoryunit operable for storing a computer program, an input mechanismoperable for inputting data into said computer system, an outputmechanism for presenting information to a user, a bus coupling theprocessor to the memory unit, input mechanism and output mechanism,wherein the mobile computer system includes various executable programmodules stored thereon where when executed are operable to performfunctions.

The computer system can comprise an in vivo monitoring module stored ona mobile computer where when executed records to the memory unit of saidmobile computer the occurrence and timing of user events comprising theopening and closing of applications resident on said mobile computer,the characters inputted in said applications, and the touch-screengestures made on said applications. The type of the event and both thefrequency and timing of the events and the various qualitative metricsregarding the event is recorded. A transmission module can also bestored on said mobile computer where when executed transmits through thewireless network interface the recordings stored in the memory unit tosaid second computer. A cognitive function module can be stored on saidsecond computer where when executed learns a function mapping from saidtransmitted recording to measurements of cognitive function using a lossfunction to determine relevant features (including frequency, timing andqualitative metrics) in said recording, identifies a set of optimalweights that produce a minimum for said loss function, and creates saidfunction mapping using said optimal weights. A plan module can also bestored on said second computer where when executed applies said functionmapping to a new transmitted recording of the occurrence and timing ofevents comprising the opening and closing of applications resident onsaid device, the characters inputted in said applications, and thetouch-screen gestures made on said applications, to calculate a newcognitive function value.

Another implementation of the invention can be a computer systemcomprising a mobile computer including a wireless network interfacewhich communicates with a second computer including a network interface,each computer further including a processor, a memory unit operable forstoring a computer program, an input mechanism operable for inputtingdata into said computer system, an output mechanism for presentinginformation to a user, a bus coupling the processor to the memory unit,input mechanism and output mechanism, wherein the mobile computer systemincludes various executable program modules stored thereon where whenexecuted are operable to perform functions.

The computer system can comprise a motion module stored on said mobilecomputer where when executed records to the memory unit of said mobilecomputer one of global positioning system longitude and latitudecoordinates, accelerometer coordinates, and gyroscope coordinates. Inanother implementation, the computer system can comprise a social modulestored on said mobile computer where when executed records to the memoryunit of said mobile computer one of outgoing and incoming phone calls,outgoing and incoming emails, and outgoing and incoming text messages.In another implementation, the computer system can comprise a learningmodule stored on said mobile computer where when executed records to thememory unit of said mobile computer one of URLs visited on an internetbrowser application, books read on an e-reader application, games playedon game applications. In another implementation, the computer system cancomprise a diet module stored on said mobile computer where whenexecuted records nutritional content of food consumed.

In any of the preceding implementations, a transmission module can bestored on said mobile computer where when executed transmits through thewireless network interface the recordings stored in the memory unit tosaid second computer. In any of the preceding implementations, anattribution module can be stored on said second computer where whenexecuted learns a function mapping from said transmitted recording tomeasurements of cognitive function using a loss function to identify aset of optimal weights that produce a minimum for said loss function,and creates said function mapping using said optimal weights. In any ofthe preceding implementations, a plan module can be stored on saidsecond computer where when executed presents to the user the attributionto said cognitive function measurements that are explained by saidrecordings.

The details of the invention and various embodiments can be betterunderstood by referring to the figures of the drawing. FIG. 1illustrates an implementation of the functional description of thesystem configured in accordance with the present invention and is notintended to limit scope as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 1 an individual uses his or her mobile device 101. In 103, theindividual taps the touch-screen to open an application, usestouch-screen gestures to scroll within an application, types messages,directions, and other content on a keyboard exposed by the applicationor uses voice commands to do the same, reads and scrolls throughcontent, and makes calls or listens to messages. The user's activity in103 is recorded on the device's persistent storage 113. In 105, the usermakes or receives calls, email messages, and text messages.

The date and time of that activity, the activity duration, and thesender and recipient phone number or email address are recorded in thedevice's persistent storage 113. The frequency of each event is recordedand the various qualitative characteristics of each event are alsorecorded. In 107, the user carries the mobile device with them whiledriving, using public transportation, walking, biking or running. Whileengaging in these activities, the mobile device's global positioningsystem (GPS) records the user's longitude and latitude coordinates in107. Similarly, in 107, the device's acceleration and gyroscopic motionalong a 3-coordinate system is recorded. The type of locations to whichthe individual traveled can be determined and the characteristic of themotion of the user can also be evaluated for fluidity or erratic motion.This information is recorded in the device's persistent storage 113. In109, the user browses URLs on an Internet browser application residenton the mobile device, or reads an e-book on an e-book reader resident onthe mobile device. The URLs browsed and the pages of the e-book read,the start time and end time between URLs and pages are recorded by 109and persisted in 113.

Note that all gesture activity, typing activity, and voice commandactivity during the use of applications tracked in 105, 107, and 109 iscaptured separately in 103 and recorded with the time and application inwhich that activity took place. In this way, the system tracks gesturesused during browsing, paging and scrolling, for example. Lastly, in 111a bar code scanning application resident on the mobile device enablesthe user to scan grocery purchases, and meals and beverages purchasedwhen the latter have bar codes. The bar code scanning application hasaccess to a database of nutritional facts. If the nutritional fact forthe scanned bar code is not in the database, then in 111 the applicationinstructs the user to photograph the nutritional fact label of the item.The bar code information and any photographs are persisted in 113.

The data captured in the device's persistent storage 113 is transmittedto cloud computers 115. Transmission uses a secure channel and can usehypertext transfer protocol secure (HTTPS) to securely transfer the datato 115. The cloud computers 115 analyze the recorded data againsthistorical recordings and against recordings averaged over other usersdemographically matched to the existing user. The output of the analysisis an evaluation of cognitive function and the attribution to changes inbehavioral activities inferred from the activities recorded in 105-111including social engagement 105, physical activity 107, learningactivity 109, and diet 111. The user in 117 logs into his or herpersonal password protected online account to view the results of theanalysis in 115 together with suggestions for improvement.

In one implementation of the method and system, in order to establish abaseline of data, supervised benchmark testing can be conducted on aninitial test group of individuals where these individuals take aneuropsychological benchmark test for cognitive function and the data isstored. Each of the same individuals who are tested can be provided withmobile devices having the computer program for performing the system andmethod. Data for each individual can be recorded as outlined herein andthe data from the mobile device usage can be correlated to the benchmarktesting results and cognitive function. Cognitive function levels andbands can also be determined from the result. Once certain baselineshave been established and correlations are made between cognitivefunction and mobile device usage, all subsequent mobile device usage byindividuals can be utilized to improve the system and method as learningoccurs. The learning from the subsequent mobile device usage can beconsidered unsupervised learning.

FIG. 2 illustrates an implementation of the functional description ofthe system configured in accordance with the present invention and isnot intended to limit scope as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 2, a user's interaction with a mobile device is captured andrecorded by the in vivo monitoring module 400. The biometric module 500confirms that the user's interactions captured by the in vivo monitoringmodule 400 truly correspond to the mobile device owner and appropriatelymarks the recordings made by that module. The transmission module 600 isresponsible for transmitting the recorded data to a cloud computerwithin 24-hours of the recording and more regularly when possible usingbroadband WiFi for the transmission or other transmission meansincluding 4G LTE transmission provide by subscriber data plans. Thismodule is also responsible for securing an encrypted channel beforetransmitting the recorded data.

The cognitive function module 700 extracts base features andhigher-order features from the data, learns a predictive model ofcognitive function, and makes cognitive function assessments of the userfrom new inputs recorded by the in vivo monitoring module 400. Module700 also computes the attribution of the user's behavioral activitiesinferred from recordings made by module 400 to the user's cognitivefunction assessments. The plan module 800 provides an online loginaccount for the user and designated individuals to review current andhistorical cognitive function trends, behavior activity attribution tothose trends, how well the user is tracking to target behavior activityand target cognitive function, and further enables the user to updatethose targets.

FIG. 3 illustrates an implementation of the computing environment of thesystem configured in accordance with the present invention and is notintended to limit scope as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 3, a user downloads and installs a software application to theirmobile device 301 from a computer in the cloud 305. The softwareapplication comprises the in vivo monitoring module 400, the biometricmodule 500, and the transmission module 600. These three modules readand write to the mobile device's persistent storage 303. The in vivomonitoring module 400 writes raw activity data to the persistent storage303. The biometric module 500 detects the new activity data and computesa biometric signature from the data. lithe biometric signature does notmatch the user's biometric signature computed from historical data, themodule requests the user to enter identifying credentials into a pop-updisplayed on the mobile device. If the user fails to confirm theiridentity, or cancels, the recorded data is marked as “non-user”generated data.

The transmission module 600 transmits the data persisted in storage 303with the annotation of the data as user or non-user generated that iscomputed by module 500 to the cloud computers 305. The cognitivefunction module 700 resides in the cloud computers 305 and computes acognitive function value from the user's mobile device usage and furthercomputes the attribution of behavioral activity, inferred from mobiledevice usage, to cognitive function value changes. The module thencompares the user's cognitive function value and inferred behavioralactivity to the behavioral norm of the user's prior data, and againstplan targets set by the plan manager 307 in the plan module 800. Theresults of the cognitive function module 700 are recorded in the planmodule 800 also residing in the cloud computers 305 and are madeavailable for review by the user and the plan manager 307. The planmanager 307 is appointed by the user and may include the user, membersof the user's family, and healthcare providers.

FIG. 4 illustrates an embodiment of the in vivo monitoring moduleconfigured in accordance with the present invention and is not intendedto limit scope as one of ordinary skill would understand on review ofthis application that other configurations could be utilized withoutdeparting from the scope of the claimed invention. Referring to FIG. 4,a mobile device such as smartphone, a tablet computer, and other similardevices can have many applications available to the user. Theseapplications include, but are not limited to, maps used for directionsand global positioning localization, weather, reminders, clock,newsstand configured with user selected publications, mail configured bythe user to include personal and work email, phone configured by theuser to include contacts, messages for texting, and an internet browser.

Module 401 tracks the application opened, the date-time that it wasopened, the date-time that it was closed and records that information onthe device's persistent storage. Upon opening an application, the userinteracts with the application through keyboard inputs, voice, andgestures. In 403, applications running on devices that supporttouch-screen gestures are recorded. The gestures include swiping andtapping and 403 records the application, datetime, gesture type, gestureduration and latency between gestures. In 405, keyboard entries arerecorded for applications enabling a keyboard. All keyboard entries arerecorded, which includes alphanumeric entries, backspace entries,capitalization, formatting entries, and editing entries. Module 405further records the latency and duration of each keyboard entry andother qualitative metrics. All recordings are stored in the device'spersistent storage with the application name and date-time of the entry.

Module 407 records voice during phone conversations and duringvoice-input commands. The number called, or application receiving thevoice command, and the date-time of the voice input are recordedtogether with the voice in the device's persistent storage. Module 409records outgoing and incoming email, text messages, and calls andfurther records recipient and sender email address, recipient and sendertext message phone number, and outgoing and incoming phone number. Theinformation is stored in the device's persistent storage together withthe date-time of the event.

For mobile devices equipped with a global positioning system (GPS),module 411 samples the device's location and time stamps the input. Formobile devices equipped with a gyroscope and an accelerometer, module411 further samples the three gyroscopic coordinates and the threeaccelerometer coordinates. The GPS, gyroscope, and accelerometer samplesare time stamped and recorded in the device's persistent storage.

Recordings from peripheral accessories that measure heart rate, bloodpressure, blood glucose, oxymetry, body temperature,electroencephalograms, and transmit those measurements to an applicationresident on a mobile device are persisted in module 413. The datatransmitted by those peripheral devices is time stamped. The peripheralaccessories can be used to obtain biological vital signs of theindividual, which can be used to determine if a decline in cognitivefunction data is due to a vital sign such as fatigue or lowblood-glucose levels rather than an actual decline in cognitive functionof the individual. The biological vitals can also be used as an alert ofa biological trend that will have a long term negative impact oncognitive function, such as hypertension.

Module 415 enables a barcode scanning application enhanced withnutritional fact information appended to each barcode entry, that tracksand records caloric input, nutritional content, alcohol, and caffeineconsumption. The dietary information can be used as an alert of adietary trend that will have a long term negative impact on cognitivefunction, such as high alcohol intake or high fat or cholesterol intake.

FIG. 5 illustrates an implementation of the biometric module configuredin accordance with the present invention and is not intended to limitscope as one of ordinary skill would understand on review of thisapplication that other configurations could be utilized withoutdeparting from the scope of the claimed invention. Referring to FIG. 5,the biometric module infers from the recorded activity on the mobiledevice whether the activity was initiated by the user or by someoneelse. As others may access the user's device for spurious reasons,correctly inferring true from false user activity is necessary.

In 501, sending an email from the mobile device, making a call from themobile device, or sending a text message from the mobile device confirmsuser identity. 501 then passes control to 507, which accepts the eventrecordings as user-generated. If 501 does not detect said activity, thenit passes control to 503 which evaluates the newly recorded gestures andgesture timings against a plurality of gesture signatures recorded forthe user. If 503 establishes a signature match of the gestures, itpasses control to 505 to further evaluate newly recorded characterpatterns and timings against a plurality of character signaturesrecorded for the user. If 505 establishes a signature match from theobserved characters it passes control to 507 which accepts the eventsrecorded as user-generated. If either 503 or 505 fail to establish asignature match, they pass control to 509 that prompts the user forvalidation. If the user validates correctly, 509 passes control to 507,and if not, control is transferred to 511 and the recorded events arerejected.

FIG. 6 illustrates an implementation of the transmission module of thesystem configured in accordance with the present invention and is notintended to limit scope as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 6, the transmission module is a background process that runs on themobile device and sleeps until it is awaken by the recording of newevents on the device in 601. Upon recording new activity, 601 passescontrol to 603 that attempts to establish WiFi access. If 603 succeeds,then it passes control to 607 to initiate transmission of all recordedactivity since the last successful transmission. If 603 fails then itpasses control to 605 of the transmission module which evaluates whethera successful transmission occurred within a 24 hour period. If 605determines that no successful transmission has occurred within a 24 hourperiod, it passes control to 607 to initiate transmission of allrecorded activity since the last successful transmission. If, however,605 confirms that a successful transmission has occurred within 24hours, it returns control to 603.

FIG. 7(a)-(d) illustrate an embodiment of the cognitive function moduleconfigured in accordance with the present invention and is not intendedto limit scope as one of ordinary skill would understand on review ofthis application that other configurations could be utilized withoutdeparting from the scope of the claimed invention. Referring to FIG.7(a), the cognitive function module accesses the recordings made by thein vivo monitoring module and transmitted to the cloud computers 305 ofFIG. 3. Patterns in a user's interactions with a mobile device capturedin 701-707 are analyzed for changes in cognitive function. In 701,changes in applications opened and closed by a user, frequency andlatencies between opening and closing, and their diurnal and weeklyvariations are inputs to the feature extraction, learning, andcomputation of cognitive function illustrated in FIGS. 7(b) and 7(c). In703, changes in a user's gestures on a touch-screen such as type ofgesture, gesture durations, including false positive gestures ofexcessive scrolling during search and excessive paging during browsing,are additional inputs to the feature extraction, learning, andcomputation.

Similarly, in 705 character inputs, recurring spelling mistakes,omissions, excessive backspace corrections, irregular latency variancesin common words, length of messages, and message coherence are inputsinto the feature extraction, learning, and computation of cognitivefunction. Lastly, in 707 signal processing of speech and voice providesadditional input to the computation including emerging irregularities inphones and phoneme latencies, and narrowing or shifting of the voicefrequency spectrum. The time of day and day of week are captured in therecordings of 701-707 and used by the cognitive function computation toadjust, or explain variances that can be attributed to individualfatigue and other factors that have short-term effects on cognitivefunction.

Further, to correct for motion artifact such as from driving or walking,GPS, gyroscope and accelerometer recordings made in 411 are used asadditional inputs in the feature extraction, learning and computation ofcognitive function. Lastly, to correct for physiologic effects such asanxiety, general malaise, illness, the physiologic measurements ofheart-rate, blood oxymetry, and body temperature when available andrecorded in 413, are used as further inputs to the evaluation ofcognitive function.

In 709-715, behavioral activities recorded on the mobile device areanalyzed to explain changes in the cognitive function that is computedusing inputs 701-707. In 709, a user's incoming and outgoing email,phone calls, and text messages, their frequencies and length are used asa proxy for the user's level of social engagement. In 711, a user'sdaily travel, the inferred mode of travel including vehicle, bicycle,foot or other, the user's sleep and rest patterns inferred by mobiledevice “quiet” times, are used to infer physical activity. In 713, whenthis data is available and when correlated with physical activity datain 711, rapid heart rates from anxiety or illness are distinguished fromexercise induced changes, improving the inference of physical activityand quantifying the intensity of that activity. In 715, analysis ofpatterns in groceries purchased, food and drinks consumed provides aproxy to nutritional intake.

FIGS. 7(b) and 7(c) illustrate an implementation of the featureextraction, learning, and computation of cognitive function configuredin accordance with the present invention, and is not intended to limitscope as one of ordinary skill would understand on review of thisapplication that other configurations could be utilized withoutdeparting from the scope of the claimed invention. Referring to FIG.7(b), mobile device recordings in 717 are encoded in the input vector ofthe visible layer of a deep belief network. The inputs in 717 comprisethe mobile usage recorded in 401-405 and if available, further comprisevoice recordings 407. In 719 the first layer of features is learned froma stochastic gradient search that minimizes the negative log-likelihoodfunction of the input vector. In 721, if higher unlearned layers exist,control is passed back to 719 and another layer of higher-order featuresis learned using as input the values from the lower hidden layer. If nofurther layers exist, 721 passes control to 723, where the weightslearned in 719 are fine-tuned using back-propagation of the errorsbetween the output from an activation function applied to the lasthidden layer with inputs 717 and the cognitive function measureassociated with those inputs.

Referring to FIG. 7(c), once the cognitive function computational modelis fully trained in 717-723, future recordings of device usage 725 isinput into the visible layer of the cognitive function computationalmodel 727. An up-propagation of the input generates the lasthidden-layer output h^((L)) that is inputted into the activationfunction in 729 to generate a new cognitive function value.

FIG. 7(d) illustrates an implementation of the attribution moduleconfigured in accordance with the present invention, and is not intendedto limit scope as one of ordinary skill would understand on review ofthis application that other configurations could be utilized withoutdeparting from the scope of the claimed invention. Referring to FIG.7(d), 731 stores the repeated evaluations of cognitive functionillustrated in FIG. 7(c) and the recordings of each user's behavioralactivity inferred from the mobile device illustrated in 709-715. In 733,consistent estimator of the parameters of a random effects regression isapplied to the data in 731. The random effects regression with theparameters computed in 733 is used to compute the variance in cognitivefunction values that is explained by the behavioral data in 731.Further, 735 computes the attribution of the behavioral activities409-415 to changes in cognitive function values stored in 731.

FIG. 8(a) illustrates an embodiment of the functional description of theplan module configured in accordance with the present invention and isnot intended to limit scope as one of ordinary skill would understand onreview of this application that other configurations could be utilizedwithout departing from the scope of the claimed invention. Referring toFIG. 8(a), the user creates and accesses an online account to obtainaggregated and detailed views of recorded data, cognitive functionevaluation and target recommendations to improve cognitive function. In801, the user creates an account and inputs personal demographic dataand health data. The user also sets privacy levels. In 803, the useraccesses recent and historical views of aggregated recordings of (i)mobility, (ii) activity, (iii) social, (iv) learning, (v) diet, (vi)applications used, (vii) electroencephalogram (EEG), and (viii)physiology. A calendar widget enables the user to specify theaggregation period to use for the recent view and the historical views.Each group of recordings offers granular views of behavior. Activityreports on physical activity and intensity, social engagement reports onoutbound and inbound number of distinct people called, emailed, or textmessaged, learning engagement reports on URLs visited and ebooks read,subject type and language, diet reports on food and beverages consumedand nutritional facts, applications reports on type of application,application name, duration and frequency of use, EEG reports onelectroencephalogram recordings, and physiology reports on heart rate,pulse oxymetry, and body temperature by time of day.

805 presents a time-series of the cognitive function evaluationscomputed in 725-729. If neuropsychological evaluations of cognitivefunction are available these are overlaid on the time-series in 805.Functional measures using blood-oxygen level dependent (BOLD) functionalmagnetic resonance imaging (fMRI) and structural volume estimates usingMRI of brain regions responsible for motivation, memory, learning thatinclude by way of example the cingulate cortex, hippocampus, andentorhinal cortex, are further overlaid on the time-series of 805.

The attribution report in 807 computes the contribution of behavioralactivity inferred from the recordings in 409-415 to changes in cognitivefunction evaluated in 725-729. In 809, the attribution report is used toset optimal target levels for mobility, physical, social, learning,diet, and physiology to restore and improve cognitive function. To helpthe user achieve those targets, information is provided in 811 thatdirects the user to relevant groups, forums, and educational material.The cycle then repeats itself.

After a target interval of one or two weeks has passed, new behavioraltrend reports are published in 803 and compared against the set targets,new values are computed for cognitive function and plotted in 805 todetermine whether the target cognitive function was attained, newattributions are evaluated in 807 to explain observed changes incognitive function, and new targets are set in 809.

FIG. 8(b) illustrates an example behavioral profile report configured inaccordance with the present invention and is not intended to limit scopeas one of ordinary skill would understand on review of this applicationthat other configurations could be utilized without departing from thescope of the claimed invention. Referring to FIG. 8(b), the exampleprofile report for social engagement 813 displays on the right, a chordchart with the number of outgoing calls made, incoming calls received,incoming voicemails received, and incoming voice mails listened to forthe month of September 2013. On the left, a chord chart displays thesame profile for the user but for the month of September one year ago in2012. This chord chart reveals greater social engagement as measured bycalls made and received in September a year ago than for September inthe present year.

The view displayed in 813 is set to the Phone Calls tab. Tabbing to theEmails and Text Messages tab would display similar chord charts for thevolume of outgoing and incoming emails and text messages made, received,and opened for the selected time period, their length and volume to orfrom an email address or phone number, and a comparison chord chart forthe same time period one year ago.

FIG. 8(c) illustrates an example attribution report configured inaccordance with the present invention and is not intended to limit scopeas one of ordinary skill would understand on review of this applicationthat other configurations could be utilized without departing from thescope of the claimed invention. Referring to FIG. 8(c), the exampleattribution report 815 displays a parallel coordinate chart thathighlights the user's behavioral changes that contribute to changes inmeasured cognitive function. Each trajectory in the parallel coordinatechart represents a daily aggregate of the user's social, mobility,activity, learning, and diet behavior. Marked deviations in thesetrajectories capture daily deviations in behavior. In 815, the tabsSocial Attribution, Mobility Attribution, Activity Attribution, LearningAttribution, and Diet Attribution display the attribution of social,mobility, activity, learning, and diet behavioral changes to changes inthe user's cognitive function.

FIG. 9(a)-(c) illustrate an embodiment of the key stroke, gesture, andGPS log data encoded unobtrusively by the in vivo monitoring moduleconfigured in accordance with the present invention and is not intendedto limit scope as one of ordinary skill would understand on review ofthis application that other key stroke, gesture, and GPS log data can beencoded unobtrusively without departing from the scope of the claimedinvention. Referring to FIG. 9(a), 901 is key log data captured by thein vivo monitoring module and transmitted to a cloud computer. Thesession_id column defines the user's session that is unique to theapplication he/she used when keying “hello world”. To determine whichapplication was used we review the app log data to find the applicationthat was launched at the time the key log entries were made. In thisparticular instance, the entry was made into the text messagingapplication. In 901, a key_type of value one represents an alphanumerickey and two represents a control key. The ASCII key code is captured inthe third column of 901 and the key description is captured in thefourth column. The fifth, sixth, and seventh columns of 901 capture thekey press time, the key release time and the key press duration timecomputed as the difference between the key press and key release times.In addition, the key latency time, not shown, can be computed as thedifference between the key_press_time of the key and thekey_release_time of the prior key.

Referring to FIG. 9(b), 903 is gesture log data captured by the in vivomonitoring module and transmitted to a cloud computer. Each gesturegenerates a row of data in 903. The application that receives thegesture is captured in the first column, app_pkg_name. An applicationcan have multiple views, and the second column in 903, view_name_jn_pkg,captures the particular view that receives the gesture. In 903, gestureswere applied to the email inbox ListView. The third column, scroll_unit,has value one if the scroll units are items and value two if the scrollunits are in pixels. In 903, the scroll unit for the email inboxListView is items. The item_count column shows thirty items in 903meaning that the inbox on that mobile device had a total of thirtyemails available for scrolling. The next two columns, from_jdx andto_jdx, capture the items in the list of thirty that were visible at theend of the gesture. At the end of the first gesture shown in the firstrow in 903, emails two through ten were visible and by the end of thelast gesture shown in the last row in 903, emails 21 through 29 werevisible. The following three columns in 903 are scroll_direction,max_scroll, and scroll_value. These columns are non-zero only when thescroll unit is pixels. The last column in 903 is the date and time thegesture was made.

Referring to FIG. 9(c), 905 is GPS log data captured by the in vivomonitoring module and transmitted to a cloud computer. Each row in 905captures the GPS latitude, longitude, altitude (in meters above sealevel), bearing (ranging from 0 to 360 degrees with North at 0 degrees),and speed (in meters per second). The date and time at which the GPScoordinates and data was captured is in the timestamp column in 905. Theaccuracy column represents the radius, in meters, of the 68% confidenceinterval for the latitude and longitude location. In 905, the datareveals that the user traveled South on Interstate 280 from SanFrancisco and took the El Monte exit off Interstate 280. The usertraveled at speeds ranging from 63.9 mph to 77.2 mph until the userreached the Page Mill exit on Interstate 280 where the user's speeddecreased to 12.3 mph. In 905, GPS measurements are taken everyfive-minute.

An individual on any given day of the week uses his or her mobile deviceregularly as a common way of communicating to others, as a form ofentertainment and as a means of retrieving and storing information.Therefore, the technology as disclosed and claimed herein does notrequire supervision over the individual users and in order to obtainmeaningful information that is useful for measuring cognitive functionand the individual does not have to change his/her lifestyle for theinformation to be gathered. For example, an individual may select anapplication on a mobile computer to launch the application by tapping onan icon displayed as part of the user interface where the user taps thetouch-screen to open an application. The individual can launch anapplication such as a weather application to check the weather forecast,launch an email application to check the email, scroll through emailsand read and respond to some emails by keying in text with some typingerrors and some misspellings. The user may get up for an early morningwalk or run or bike ride and a combination of information may becaptured by the global position function of the device as well asinformation or data from the gyroscopic motion sensor of the deviceindicative of the user exercising. When walking the user may trip orstumble quite often or their motion may not be fluid, which may becapture by the accelerometer and gyroscopic motion sensor as a suddenand violent movement. Also the user may actually trip and fall. The usermay go for a drive over what appears to be a familiar route, but the GPSfunction captures what appears to be a series of incorrect turns. Thismay be indicative of a problem, although it may also be captured thatthe individual is also placing a call, on a call, or sending a text atthe same time that the false or incorrect turn was made. The user canopen a navigation application and enter a location search query and canuse touch-screen gestures to scroll around on map within an applicationand select items on the map for additional feature information. Whenscrolling through the map, the individual can touch and swipe the map tofollow highlighted directions or to search for a point of interest.Touches that dwell too long or swipes that are too short may result inan undesired response from the mobile computer. During the day, theindividual may be traveling to and fro solo, but may need to reach outto their spouse, or son or daughter to coordinate a pick-up or meeting.The user can type text messages to selectively reach out. The individualmay continue a text dialogue for a period of time. During the textdialogue the individual may have to back space several times to correcta typing error. The individual may want to meet with their spouse fordinner, but they do not know exactly were the restaurant of choice islocated. The user can use the voice command feature to requestdirections and the GPS can be tracking the user's location, the user canrequest other content by keying in on a keyboard presented by theapplication or use voice commands to do the same, read and scrollthrough content, and the user can make calls or listen to messages. Theuser's activity is recorded on the device's persistent storage. The usermakes or receives calls, email messages, and text messages. There canalso be several days in a row where the user is carrying his/her mobiledevice, but out of the ordinary, the user may not place any call or sendany text messages, browse their favorite social media application orvisit their favorite website.

The date and time of each of the above activities, the activityduration, and the sender and recipient phone number or email address canbe recorded in the device's persistent storage. The frequency of eachevent is recorded and the various qualitative characteristics of eachevent are also recorded. The user carries the mobile device with themwhile driving, using public transportation, walking, biking or running.While engaging in these activities, the mobile device's globalpositioning system (GPS) records the user's longitude and latitudecoordinates. Similarly, during those activities the device'sacceleration and gyroscopic motion along a 3-coordinate system isrecorded. The type of locations that the individual traveled can bedetermined and the characteristic of the motion of the user can also beevaluated for fluidity or erratic motion. This information is recordedin the device's persistent storage. The user browses URLs on an internetbrowser application resident on the mobile device, or reads an e-book onan e-book reader resident on the mobile device. The URLs browsed and thepages of the e-book read, the start time and end time between URLs andpages are recorded by the system and method of this invention andpersisted in the device's persistent storage.

Note that all gesture activity, typing activity, and voice commandactivity during the use of applications is captured separately in andrecorded with the time and application in which that activity tookplace. In this way, the system tracks gestures used during browsing,paging and scrolling, for example. Lastly, a bar code scanningapplication resident on the mobile device enables the user to scangrocery purchases, and meals and beverages purchased when the latterhave bar codes. The bar code scanning application has access to adatabase of nutritional facts. If the nutritional fact for the scannedbar code is not in the database, then the application instructs the userto photograph the nutritional fact label of the item. The bar codeinformation and any photographs are persisted in the device's persistentstorage.

The data captured in the device's persistent storage can be transmittedto cloud computers. Transmission can use a secure channel and can usehypertext transfer protocol secure (HTTPS) to securely transfer thedata. The cloud computers can analyze the recorded data againsthistorical recordings and against recordings averaged over other usersdemographically matched to the existing user. The output of the analysisis an evaluation of cognitive function and the attribution to changes inbehavioral activities inferred from the activities recorded in includingsocial engagement, physical activity, learning activity, and diet. Theuser logs into his or her personal password protected online account toview the results of the analysis together with suggestions forimprovement.

Various recordings of data representative of the quantity and quality ofinteractions and occurrences of events using the mobile computing devicecan be made. The data can be encoded with representative tags.

A description of the type of encoded data with representative logsemantics when an application is launched on the mobile device is asfollows:

App Log Description app_pkg_name Application launched app_start_timeStart date and time of use app_end_time End date and time of use

A description of the type of encoded data with representative logsemantics following incoming, outgoing and missed calls on the mobiledevice is as follows:

Call Log Description call_type Values (1, 2, 3) meaning (incoming,outgoing, missed) phone_number Phone number of call call_start_timeStart date and time of call call_end_time End date and time of callaudio_recording_file File name of recorded call and null if not recorded

A description of the type of encoded data with representative logsemantics following gestures made on the touch-screen of the mobiledevice during use of an application is as follows:

Gesture Log Description app_pkg_name Application in use view_name_in_pkgPackage within application (eg. Contacts or missed calls, in phoneapplication) scroll_unit Values (1, 2) meaning (items, pixels)item_count Number of scrollable items from_idx Index of first itemvisible when scrolling to_idx Index of last item visible when scrollingscroll_direction Values (1, 2) meaning scroll in (X, Y) directionmax_scroll Max_scroll x dir (or y dir): gives the max scroll offset ofthe source left edge (or top edge) in pixels scroll_value Scroll_value Xdir (or Y dir): gives the scroll offset of the source left edge (or topedge) in pixels timestamp Date and time

A description of the type of encoded GPS data with representative logsemantics on the mobile device is as follows:

GPS Log Description latitude Latitude of current location longitudeLongitude of current location altitude Altitude of current locationbearing Horizontal direction of travel in degrees (0.0, 360.0] speedTravel speed in meters per second accuracy Radius in meters of 68%confidence circle timestamp Date and time

A description of the type of encoded data with representative logsemantics when the keyboard is used on the mobile device is as follows:

Key Log Description session_id Identifies each session key_type Values(1, 2) meaning (alphanumeric, control key) key_code Ascii code ofpressed key key_desc Either the alphanumeric value or a controldescriptor key_press_time Date and time in milliseconds when key ispressed key_release_time Date and time in milliseconds when key isreleased key_press_duration Duration of key press in milliseconds

A description of the type of encoded sensor data with representative logsemantics on the mobile device is as follows:

Sensor Log Description sensor_type Values (1, 2) meaning (accelerometer,gyroscope) value_0 Value of first coordinate measurement value_1 Valueof second coordinate measurement value_2 Value of third coordinatemeasurement timestamp Date and time of coordinate measurements

A description of the type of encoded data with representative logsemantics when text messages are sent or received on the mobile deviceis as follows:

SMS Log Description sms_type Values (1, 2) meaning (Incoming, Outgoing)phone_number Phone number of message message Message text timestamp Dateand time of message

A description of the type of encoded data with representative logsemantics when URLs are browsed on the mobile device is as follows:

URL Log Description URL URL viewed timestamp Start date and time of URLview

In one implementation of the method and system, in order to establish abaseline of data, supervised benchmark testing can be conducted on aninitial test group of individuals where these individual take aneuropsychological benchmark test for cognitive function and the data isstored. Each of the same individuals who are tested can be provided withmobile devices having the computer program for performing the system andmethod. Data for each individual can be recorded as outlined herein andthe data from the mobile device usage can be correlated to the benchmarktesting results and cognitive function. Cognitive function levels andbands can also be determined from the result. Once certain baselineshave been established and correlations are made between cognitivefunction and mobile device usage, all subsequent mobile device usage byindividuals can be utilized to improve the system and method as learningoccurs. The learning from the subsequent mobile device usage can beconsidered unsupervised learning.

The cognitive function module accesses the recordings made by the invivo monitoring module and transmitted to the cloud computers. Patternsin a user's interactions with a mobile device captured are analyzed forchanges in cognitive function. The above described activity for anindividual and his interface with a mobile computing device can becaptured and utilize. Changes in applications opened and closed by auser, frequency and latencies between opening and closing, and theirdiurnal and weekly variations are inputs to the feature extraction,learning, and computation of cognitive function. Changes in a user'sgestures on a touch-screen such as type of gesture, gesture durations,including false positive gestures of excessive scrolling during searchand excessive paging during browsing, are additional inputs to thefeature extraction, learning, and computation.

Similarly, when the above described individual makes character inputs,recurring spelling mistakes, omissions, excessive backspace corrections,irregular latency variances in common words, length of messages, andmessage coherence can all be inputs into the feature extraction,learning, and computation of cognitive function. Signal processing ofspeech and voice provides additional input to the computation includingemerging irregularities in phones and phoneme latencies, and narrowingor shifting of the voice frequency spectrum. The time of day and day ofweek are captured in the recordings of and used by the cognitivefunction computation to adjust, or explain variances that can beattributed to individual fatigue and other factors that have short-termeffects on cognitive function.

Further, to correct for motion artifact such as from driving or walking,GPS, gyroscope and accelerometer recordings made are used as additionalinputs in the feature extraction, learning and computation of cognitivefunction. To correct for physiologic effects such as anxiety, generalmalaise, illness, the physiologic measurements of heart-rate, bloodpressure, blood glucose, blood oxymetry, and body temperature whenavailable and recorded are used as further inputs to the evaluation ofcognitive function. Behavioral activities recorded on the mobile deviceare analyzed to explain changes in the cognitive function that iscomputed using inputs. A user's incoming and outgoing email, phonecalls, and text messages, their frequencies and length are used as aproxy for the user's level of social engagement. A user's daily travel,the inferred mode of travel including vehicle, bicycle, foot or other,the user's sleep and rest patterns inferred by mobile device “quiet”times, are used to infer physical activity. When this data is availableand when correlated with physical activity data such as rapid heartrates from anxiety or illness are distinguished from exercise inducedchanges, improving the inference of physical activity and quantifyingthe intensity of that activity. Analysis of patterns in groceriespurchased, food and drinks consumed provides a proxy to nutritionalintake is also made.

The various implementations and examples shown above illustrate a methodand system for assessing cognitive function using a mobile computingdevice. A user of the present method and system may choose any of theabove implementations, or an equivalent thereof, depending upon thedesired application. In this regard, it is recognized that various formsof the subject method and system could be utilized without departingfrom the spirit and scope of the present implementation.

As is evident from the foregoing description, certain aspects of thepresent implementation are not limited by the particular details of theexamples illustrated herein, and it is therefore contemplated that othermodifications and applications, or equivalents thereof, will occur tothose skilled in the art. It is accordingly intended that the claimsshall cover all such modifications and applications that do not departfrom the spirit and scope of the present implementation. Accordingly,the specification and drawings are to be regarded in an illustrativerather than a restrictive sense.

Certain systems, apparatus, applications or processes are describedherein as including a number of modules. A module may be a unit ofdistinct functionality that may be presented in software, hardware, orcombinations thereof. When the functionality of a module is performed inany part through software, the module includes a computer-readablemedium. The modules may be regarded as being communicatively coupled.The inventive subject matter may be represented in a variety ofdifferent implementations of which there are many possible permutations.

The methods described herein do not have to be executed in the orderdescribed, or in any particular order. Moreover, various activitiesdescribed with respect to the methods identified herein can be executedin serial or parallel fashion. In the foregoing Detailed Description, itcan be seen that various features are grouped together in a singleembodiment for the purpose of streamlining the disclosure. This methodof disclosure is not to be interpreted as reflecting an intention thatthe claimed embodiments require more features than are expressly recitedin each claim. Rather, as the following claims reflect, inventivesubject matter may lie in less than all features of a single disclosedembodiment. Thus, the following claims are hereby incorporated into theDetailed Description, with each claim standing on its own as a separateembodiment.

In an example embodiment, the machine operates as a standalone device ormay be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a server computer, a client computer, a personal computer(PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant(PDA), a cellular telephone, a smart phone, a web appliance, a networkrouter, switch or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine or computing device. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein.

The example computer system and client computers include a processor(e.g., a central processing unit (CPU) a graphics processing unit (GPU)or both), a main memory and a static memory, which communicate with eachother via a bus. The computer system may further include avideo/graphical display unit (e.g., a liquid crystal display (LCD) or acathode ray tube (CRT)). The computer system and client computingdevices also include an alphanumeric input device (e.g., a keyboard ortouch-screen), a cursor control device (e.g., a mouse or gestures on atouch-screen), a drive unit, a signal generation device (e.g., a speakerand microphone) and a network interface device.

The drive unit includes a computer-readable medium on which is storedone or more sets of instructions (e.g., software) embodying any one ormore of the methodologies or systems described herein. The software mayalso reside, completely or at least partially, within the main memoryand/or within the processor during execution thereof by the computersystem, the main memory and the processor also constitutingcomputer-readable media. The software may further be transmitted orreceived over a network via the network interface device.

The term “computer-readable medium” should be taken to include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or more sets ofinstructions. The term “computer-readable medium” shall also be taken toinclude any medium that is capable of storing or encoding a set ofinstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the presentimplementation. The term “computer-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical media, and magnetic media.

The invention claimed is:
 1. A computer-implemented means for assessingbrain health of a person based on the person's use of a mobile device,wherein the computer-implemented means comprises a processing means, andwherein: the computer-implemented means receives inputs from amonitoring means installed on the mobile device, the inputs associatedwith one or more interactions of the person with the mobile device, theinputs recorded by the monitoring means without requiring additionalactions by the person while the inputs are recorded; thecomputer-implemented means learns a function mapping from the receivedinputs to a brain health metric obtained from a brain test, whereinlearning the function mapping uses a loss-function to determine relevantfeatures of the inputs and to identify a set of optimal weights for thefunction mapping that produce a minimum of the loss-function; thecomputer-implemented means receives a set of new inputs from themonitoring means, the new inputs associated with one or more newinteractions of the person with the mobile device; and thecomputer-implemented means applies the learned function mapping to thenew inputs to calculate a new brain health metric.
 2. Thecomputer-implemented means of claim 1, wherein the brain test is one ofa neuropsychological test, a neurophysiological test, a magneticresonance imaging (MM) test, a functional MRI test, and a positronemission tomography test.
 3. The computer-implemented means of claim 1,wherein the function mapping is one of a linear function and anon-linear function.
 4. The computer-implemented means of claim 1,wherein the one or more interactions includes at least one of:applications opened, inputs typed, gesture patterns used on a touchscreen, body motions, eye movements, voice input, accelerometer sensordata, and gyroscopic sensor data.
 5. The computer-implemented means ofclaim 1, wherein the mobile device is one of: a smart phone, a tabletcomputer, and a wearable mobile device.
 6. The computer-implementedmeans of claim 1, wherein to quantify an effect of at least one ofsocial interaction, mobility, physiology, cognitive stimulation,behavioral therapy, and diet on the brain health metric, thecomputer-implemented means: a. learns an attribution function mappingfrom measurements of at least one of social interaction, mobility,physiology, cognitive stimulation, behavioral therapy, and diet to thebrain health metric, the measurements associated with the inputs,wherein learning the attribution function uses a loss-function todetermine relevant features of the inputs and to identify a set ofoptimal weights for the attribution function that produce a minimum ofthe loss-function and b. outputs an attribution to the brain healthmetric that is explained by the measurements associated with the inputs.7. The computer-implemented means of claim 6, wherein thecomputer-implemented means makes a recommendation or care-planadjustment in one or more of social interaction, mobility, physiology,sleep, cognitive stimulation, and diet.
 8. A computer-implemented methodfor assessing brain health of a person based on the person's use of amobile device, the computer-implemented method executing instructions onone or more hardware processors of a computing system, the instructionscomprising the steps of: receiving, by the computing system, inputs froma monitoring module installed on the mobile device, the inputsassociated with one or more interactions of the person with the mobiledevice, the inputs recorded by the monitoring module without requiringadditional actions by the person while the inputs are recorded;learning, by the computing system, a function mapping from the receivedinputs to a brain health metric obtained from a brain test, whereinlearning the function mapping uses a loss-function to determine relevantfeatures of the inputs and to identify a set of optimal weights for thefunction mapping that produce a minimum of the loss-function; receiving,by the computing system, a set of new inputs from the monitoring module,the new inputs associated with one or more new interactions of theperson with the mobile device; and applying, by the computing system,the learned function mapping to the new inputs to calculate a new brainhealth metric.
 9. The computer-implemented method of claim 8, whereinthe brain test is one of a neuropsychological test, a neurophysiologicaltest, a magnetic resonance imaging (MM) test, a functional MRI test, anda positron emission tomography test.
 10. The computer-implemented methodof claim 8, wherein the function mapping is one of a linear function anda non-linear function.
 11. The computer-implemented method of claim 8,wherein the one or more interactions includes at least one of:applications opened, inputs typed, gesture patterns used on a touchscreen, body motions, eye movements, voice input, accelerometer sensordata, and gyroscopic sensor data.
 12. The computer-implemented method ofclaim 8, wherein the mobile device is one of: a smart phone, a tabletcomputer, and a wearable mobile device.
 13. The computer-implementedmethod of claim 8, wherein to quantify an effect of at least one ofsocial interaction, mobility, physiology, cognitive stimulation,behavioral therapy, and diet on the brain health metric, theinstructions further comprise the steps of: a. learning an attributionfunction mapping from measurements of at least one of socialinteraction, mobility, physiology, cognitive stimulation, behavioraltherapy, and diet to the brain health metric, the measurementsassociated with the inputs, wherein learning the attribution functionuses a loss-function to determine relevant features of the inputs and toidentify a set of optimal weights for the attribution function thatproduce a minimum of the loss-function; and b. outputting an attributionto the brain health metric that is explained by the measurementsassociated with the inputs.
 14. The computer-implemented method of claim13, wherein the instructions further comprise the step of making arecommendation or care-plan adjustment in one or more of socialinteraction, mobility, physiology, sleep, cognitive stimulation, anddiet.
 15. A computer-readable medium comprising instructions that whenexecuted by a processor running on a computing system perform a methodfor assessing brain health of a person based on the person's use of amobile device, the instructions comprising the steps of: receiving, bythe computing system, inputs from a monitoring module installed on themobile device, the inputs associated with one or more interactions ofthe person with the mobile device, the inputs recorded by the monitoringmodule without requiring additional actions by the person while theinputs are recorded; learning, by the computing system, a functionmapping from the received inputs to a brain health metric obtained froma brain test, wherein learning the function mapping uses a loss-functionto determine relevant features of the inputs and to identify a set ofoptimal weights for the learning function mapping that produce a minimumof the loss-function; receiving, by the computing system, a set of newinputs from the monitoring module, the new inputs associated with one ormore new interactions of the person with the mobile device; andapplying, by the computing system, the learned function mapping to thenew inputs to calculate a new brain health metric.
 16. Thecomputer-readable medium of claim 15, wherein the brain test is one of aneuropsychological test, a neurophysiological test, a magnetic resonanceimaging (MM) test, a functional Mill test, and a positron emissiontomography test.
 17. The computer-readable medium of claim 15, whereinthe function mapping is one of a linear function and a non-linearfunction.
 18. The computer-readable medium of claim 15, wherein the oneor more interactions includes at least one of: applications opened,inputs typed, gesture patterns used on a touch screen, body motions, eyemovements, voice input, accelerometer sensor data, and gyroscopic sensordata.
 19. The computer-readable medium of claim 15, wherein to quantifyan effect of at least one of social interaction, mobility, physiology,cognitive stimulation, behavioral therapy, and diet on the brain healthmetric, the instructions further comprise the steps of: a. learning anattribution function mapping from measurements of at least one of socialinteraction, mobility, physiology, cognitive stimulation, behavioraltherapy, and diet to the brain health metric, the measurementsassociated with the inputs, wherein learning the attribution functionuses a loss-function to determine relevant features of the inputs and toidentify a set of optimal weights for the attribution function thatproduce a minimum of the loss-function; and b. outputting an attributionto the brain health metric that is explained by the measurementsassociated with the inputs.
 20. The computer-readable medium of claim19, wherein the instructions further comprise the step of making arecommendation or care-plan adjustment in one or more of socialinteraction, mobility, physiology, sleep, cognitive stimulation, anddiet.